Skip to content

fix(writer): text is a vortex.varbinview unless a scheme is smaller (#583) - #677

Merged
dfa1 merged 1 commit into
mainfrom
feat/583-utf8-varbinview
Nov 6, 2026
Merged

dfa1 merged 1 commit into
mainfrom
feat/583-utf8-varbinview

Conversation

@dfa1

@dfa1 dfa1 commented Nov 6, 2026

Copy link
Copy Markdown
Owner

Stacked on #676.

  • Rust's canonical text array is vortex.varbinview, the baseline every scheme must beat; VarBinScheme is binary-only. The Utf8 competition scores against view bytes, excludes varbin for Utf8, and falls back to varbinview.
  • The varbinview node carries is_constant, null_count, view size.
  • An empty varbinview has an empty views buffer (Rust checks it; found by the list interop tests).
  • String zone tables now match vortex-jni except high, whose 3 bounds Rust's trained FSST table compresses.

Not done: is_sorted on varbin/FSST offsets and FSST lengths shape — depend on the trained symbol table.

🤖 Generated with Claude Code

https://claude.ai/code/session_011fRbYsrGtpKHp9APhU8qDV

…as Rust's canonical form (#583)

Rust's canonical text array is vortex.varbinview and the baseline every string scheme must beat;
VarBinScheme re-encodes binary only. The Utf8 competition now scores against the views' size,
leaves vortex.varbin and vortex.varbinview out as candidates for Utf8 (varbinview for binary
too), and falls back to varbinview. The node carries is_constant, null_count and the size of its
views, and an empty array's views buffer is empty, which Rust checks against the row count.
The string zone tables now match vortex-jni, bar one whose three bounds Rust's trained FSST
table compresses.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011fRbYsrGtpKHp9APhU8qDV
@dfa1
dfa1 force-pushed the feat/583-utf8-varbinview branch from b8b840b to 358dda5 Compare November 6, 2026 20:35
@dfa1
dfa1 changed the base branch from feat/583-zone-table-shape to main November 6, 2026 20:35
@dfa1
dfa1 merged commit 8f88a43 into main Nov 6, 2026
6 checks passed
@dfa1
dfa1 deleted the feat/583-utf8-varbinview branch November 18, 2026 21:56
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant